Running Spark SQL tests locally with iceberg_compat. If that passes, I'll likely want to undo #1376 and try again.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 45.92%. Comparing base (f09f8af) to head (d401b98).
⚠️ Report is 691 commits behind head on main.

@@              Coverage Diff              @@
##               main    #2197       +/-   ##
=============================================
- Coverage     56.12%   45.92%   -10.20%     
- Complexity      976     1219      +243     
=============================================
  Files           119      164       +45     
  Lines         11743    13989     +2246     
  Branches       2251     2369      +118     
=============================================
- Hits           6591     6425      -166     
- Misses         4012     6525     +2513     
+ Partials       1140     1039      -101     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

You may want to undo parts of #1376 to really test the change. The changes to CometTestBase.getPrimitiveTypesParquetSchema were so that the schema would not contain unsigned types if the reader type was one of the datafusion based readers. Undoing that change will be a primary test.

So instead of getting null on the incorrectly written values we now get different values. So garbage in is still garbage out, but it's new garbage!

I seem to recall that I tested a badly written file with apache/arrow#7055 and did not get garbage. I'll take a look at it again.
Thanks for kicking off this effort @mbutrovich!

I don't think the DataFusion-backed scans will ever be fully compatible. #3238 with this PR we just have it as something someone needs to opt into.